D2 + D3: measured autotune loop + arbiter fallback log (sm_120) - #294
Conversation
D2 (measured autotune) — emit/autotune.py layers on the D1 arbiter's `measure` seam: measured_arbitrate() F4-gates the candidates, times each survivor on-device (measure_latency, median-of-N after warmup), and caches the fastest in a MeasureCache keyed by (device, target, op, shape-bucket, dtype) — measure-at- first-miss (a re-query hits the cache, no re-timing). Lead-safety holds: only in-budget F4-passing candidates are timed, so a faster-but-wrong kernel can't win. runtime._nvidia_device_name() supplies the sm_<cc> device tag. Live on sm_120: times the shipped vs emitted GEMM lanes and caches the winner per bucket. D3 (fallback log) — the arbiter records every dispatch as (target, op, selected, tag) (candidate._note_arbiter_dispatch, wired into run_arbitrated + run_measured_arbitrated); arbiter_dispatch_histogram() answers "did the compiled path win, silently degrade, or was there no candidate?" per (target, op) — a selection that ran but returned a reference tag is the silent degrade (the arbiter-layer analog of runtime.dispatch_fallback_log). Proven host-free and live (the emitted lane forced on a ragged shape it can't run logs a `degraded`). Tests: test_arbiter_autotune.py (5 host-free — measured pick+cache, bucketing, won/degraded/no_candidate, histogram filter) + 2 nvidia sm_120 live gates. Still open: persist the cache as the committed fleet-shared corpus (Theory §7.5, hangs off MeasureCache.to_dict); [AMD] gfx1151 wiring. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 628bc922b8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if c.name == rec.winner and c.applies_to(region) and c.available(): | ||
| return c |
There was a problem hiding this comment.
When a cached autotune winner is reused, this only checks region-level applicability and availability, not whether the candidate can execute the current input shape. For example, NvidiaMmaGemmEmittedCandidate.applies_to() accepts all bf16/f16 matmuls, but its run() declines to the NumPy reference when M%16, N%8, or K%16 fail; after tuning an aligned shape in a power-of-two bucket such as 32x16x32, a ragged shape like 24x16x32 hits the same bucket and returns the emitted candidate without reconsidering the shipped GEMM, causing run_measured_arbitrated() to silently degrade to the reference. The cache hit needs an input-shape capability check, an exact/alignment-aware key, or a revalidation that excludes reference declines.
Useful? React with 👍 / 👎.
The NVIDIA 2D GEMM helpers took M,K from A and N from B but discarded B's K, so a mismatched contraction (A is MxK, B is K2xN, K != K2) passed A's K to the C ABI — which then copies K*N elements from a B buffer holding only K2*N, overreading host memory or computing against the wrong slice instead of raising like MatmulRegion.reference / the JIT path. Both _nvidia_mma_gemm_2d and _nvidia_ptx_gemm_2d now validate rank-2 + matching K BEFORE any side effect (so a malformed matmul fails fast, host-free, ahead of the lib load). The shipped candidate previously had no such check (the emitted lane's _aligned_2d already rejected it). Host-free regression test added. Co-authored-by: angst <angstroms01@gmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
1 participant
Section B, items 2 & 3 — the measured autotune loop (D2) and arbiter fallback logging (D3), building on the matmul op-kind from #293. Verified on the live sm_120 box.
D2 — measured autotune loop (
emit/autotune.py)Layers on the D1 arbiter's existing
measureseam.measured_arbitrate():measure_latency, median-of-N after warmup),MeasureCachekeyed by(device, target, op, shape-bucket, dtype)— measure-at-first-miss (a re-query hits the cache, no re-timing).Lead-safety holds end-to-end: only F4-passing, in-accuracy-budget candidates are ever timed, so a faster-but-wrong kernel can't win.
runtime._nvidia_device_name()supplies thesm_<cc>device tag so a config measured on one device isn't reused on another.Live on sm_120: times the shipped vs emitted GEMM lanes and caches the winner per bucket (shipped 0.476ms vs emitted 0.484ms on 64³ → picks the faster by latency, not tier).
D3 — arbiter fallback log (
candidate.py)Every arbitrated dispatch records
(target, op, selected, tag)(_note_arbiter_dispatch, wired intorun_arbitrated+run_measured_arbitrated).arbiter_dispatch_histogram()answers "did the compiled path win, silently degrade, or was there no candidate?" per(target, op):The arbiter-layer analog of
runtime.dispatch_fallback_log.Verified
test_arbiter_autotune.py(5 — measured pick+cache, shape bucketing, won/degraded/no_candidate, histogram filter);degraded.30 passed / 1 skipped; ruff + mypy clean; drift green.
Still open (this section)
MeasureCache.to_dict).[AMD]gfx1151 wiring of the same loop.Section C (flash-attn sm_120; generic CUDA lane past f32) is next.
🤖 Generated with Claude Code